Home / Deep Learning

Tensorflow

• Keras Multi-layer Perceptron

  • Keras Multi-layer Perceptron (MLP) for Binary Classifications
  • Keras Multi-layer Perceptron (MLP) for Multi-Class Classifications
  • Keras Multi-layer Perceptron (MLP) for Image Classifications
  • Keras Multi-layer Perceptron (MLP) for Regression

• Tensorflow Premade Estimators

  • Tensorflow Linear Classifier
  • Tensorflow DNN Classifier
  • Tensorflow Boosted Trees Classifier
  • Tensorflow Boosted Trees Classifier with Feature Importance Analysis

• Tensorflow Image and Text Classification

  • Image Augmentations with TensorFlow
  • Image Classification with TensorFlow
  • Text Classification Pipeline with Tensorflow

Tensorflow DNN Classifier

In this article, we demonstrate solving a classification problem in TensorFlow using Estimators using the UCI ML Wine recognition dataset. This dataset also can be accessed via the scikit-learn datasets.

Dataset Information

These data are the results of a chemical analysis of wines grown in the same region in Italy but derived from three different cultivars. The analysis determined the quantities of 13 constituents found in each of the three types of wines.

import numpy as np
import pandas as pd
from sklearn import datasets

data = datasets.load_wine()
Data = pd.DataFrame(data['data'],
                    columns = [x.title().replace('_',' ').replace('Of','of').replace('Od','OD') for x in data['feature_names']])
Labels_dict = dict(zip(list(np.sort(np.unique(data['target'].tolist()))),
                       list([x.title() for x in data['target_names']])))
Target = 'Class'
Data[Target] = data['target']
display(Data)
print(data['DESCR'])

	Alcohol	Malic Acid	Ash	Alcalinity of Ash	Magnesium	Total Phenols	Flavanoids	Nonflavanoid Phenols	Proanthocyanins	Color Intensity	Hue	OD280/OD315 of Diluted Wines	Proline	Class
0	14.23	1.71	2.43	15.6	127.0	2.80	3.06	0.28	2.29	5.64	1.04	3.92	1065.0	0
1	13.20	1.78	2.14	11.2	100.0	2.65	2.76	0.26	1.28	4.38	1.05	3.40	1050.0	0
2	13.16	2.36	2.67	18.6	101.0	2.80	3.24	0.30	2.81	5.68	1.03	3.17	1185.0	0
3	14.37	1.95	2.50	16.8	113.0	3.85	3.49	0.24	2.18	7.80	0.86	3.45	1480.0	0
4	13.24	2.59	2.87	21.0	118.0	2.80	2.69	0.39	1.82	4.32	1.04	2.93	735.0	0
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
173	13.71	5.65	2.45	20.5	95.0	1.68	0.61	0.52	1.06	7.70	0.64	1.74	740.0	2
174	13.40	3.91	2.48	23.0	102.0	1.80	0.75	0.43	1.41	7.30	0.70	1.56	750.0	2
175	13.27	4.28	2.26	20.0	120.0	1.59	0.69	0.43	1.35	10.20	0.59	1.56	835.0	2
176	13.17	2.59	2.37	20.0	120.0	1.65	0.68	0.53	1.46	9.30	0.60	1.62	840.0	2
177	14.13	4.10	2.74	24.5	96.0	2.05	0.76	0.56	1.35	9.20	0.61	1.60	560.0	2

178 rows × 14 columns

.. _wine_dataset:

Wine recognition dataset
------------------------

**Data Set Characteristics:**

    :Number of Instances: 178 (50 in each of three classes)
    :Number of Attributes: 13 numeric, predictive attributes and the class
    :Attribute Information:
 		- Alcohol
 		- Malic acid
 		- Ash
		- Alcalinity of ash  
 		- Magnesium
		- Total phenols
 		- Flavanoids
 		- Nonflavanoid phenols
 		- Proanthocyanins
		- Color intensity
 		- Hue
 		- OD280/OD315 of diluted wines
 		- Proline

    - class:
            - class_0
            - class_1
            - class_2
		
    :Summary Statistics:
    
    ============================= ==== ===== ======= =====
                                   Min   Max   Mean     SD
    ============================= ==== ===== ======= =====
    Alcohol:                      11.0  14.8    13.0   0.8
    Malic Acid:                   0.74  5.80    2.34  1.12
    Ash:                          1.36  3.23    2.36  0.27
    Alcalinity of Ash:            10.6  30.0    19.5   3.3
    Magnesium:                    70.0 162.0    99.7  14.3
    Total Phenols:                0.98  3.88    2.29  0.63
    Flavanoids:                   0.34  5.08    2.03  1.00
    Nonflavanoid Phenols:         0.13  0.66    0.36  0.12
    Proanthocyanins:              0.41  3.58    1.59  0.57
    Colour Intensity:              1.3  13.0     5.1   2.3
    Hue:                          0.48  1.71    0.96  0.23
    OD280/OD315 of diluted wines: 1.27  4.00    2.61  0.71
    Proline:                       278  1680     746   315
    ============================= ==== ===== ======= =====

    :Missing Attribute Values: None
    :Class Distribution: class_0 (59), class_1 (71), class_2 (48)
    :Creator: R.A. Fisher
    :Donor: Michael Marshall (MARSHALL%PLU@io.arc.nasa.gov)
    :Date: July, 1988

This is a copy of UCI ML Wine recognition datasets.
https://archive.ics.uci.edu/ml/machine-learning-databases/wine/wine.data

The data is the results of a chemical analysis of wines grown in the same
region in Italy by three different cultivators. There are thirteen different
measurements taken for different constituents found in the three types of
wine.

Original Owners: 

Forina, M. et al, PARVUS - 
An Extendible Package for Data Exploration, Classification and Correlation. 
Institute of Pharmaceutical and Food Analysis and Technologies,
Via Brigata Salerno, 16147 Genoa, Italy.

Citation:

Lichman, M. (2013). UCI Machine Learning Repository
[https://archive.ics.uci.edu/ml]. Irvine, CA: University of California,
School of Information and Computer Science. 

.. topic:: References

  (1) S. Aeberhard, D. Coomans and O. de Vel, 
  Comparison of Classifiers in High Dimensional Settings, 
  Tech. Rep. no. 92-02, (1992), Dept. of Computer Science and Dept. of  
  Mathematics and Statistics, James Cook University of North Queensland. 
  (Also submitted to Technometrics). 

  The data was used with many others for comparing various 
  classifiers. The classes are separable, though only RDA 
  has achieved 100% correct classification. 
  (RDA : 100%, QDA 99.4%, LDA 98.9%, 1NN 96.1% (z-transformed data)) 
  (All results using the leave-one-out technique) 

  (2) S. Aeberhard, D. Coomans and O. de Vel, 
  "THE CLASSIFICATION PERFORMANCE OF RDA" 
  Tech. Rep. no. 92-01, (1992), Dept. of Computer Science and Dept. of 
  Mathematics and Statistics, James Cook University of North Queensland. 
  (Also submitted to Journal of Chemometrics).

Features with high variance

Moreover, high variance for some features can hurt our modeling process. For this reason, we would like to standardize features by removing the mean and scaling to unit variance.

from sklearn.preprocessing import StandardScaler
X = Data.drop(columns = [Target])
y = Data[Target]

scaler = StandardScaler()
X_std = scaler.fit_transform(X)
X_std = pd.DataFrame(data = X_std, columns = X.columns)
del scaler

PD = dict(figsize = (16, 5), hspace = 0.2, annot_text_size = 12, text_size = 14, word_break = 2)
from HD_DeepLearning import Feature_Normalize_Plot
Feature_Normalize_Plot(X, X_std, PD)

X_std.columns = [x.replace(' ','_').replace('/','_') for x in X_std.columns]
Feat_Dict = dict(zip(X.columns, X_std.columns))

Train and Test sets

import plotly.express as px
from HD_DeepLearning import DatasetTargetDist

Pull = [0 for x in range((len(Labels_dict)-1))]
Pull.append(.05)
PD = dict(PieColors = px.colors.sequential.BuPu[1:-1:3], TableColors = ['Navy','White'], hole = .4,
          row_heights=[0.35, 0.65],textfont = 14, height = 520, tablecolumnwidth = [0.25, 0.15, 0.15],
          pull = Pull, legend_title = Target, title_x = 0.5, title_y = 0.9, pie_legend = [0.1, 0.12])
del Pull
DatasetTargetDist(Data, Target, Labels_dict, PD)

StratifiedKFold is a variation of k-fold which returns stratified folds: each set contains approximately the same percentage of samples of each target class as the complete set.

from sklearn.model_selection import StratifiedShuffleSplit

def HD_StratifiedShuffleSplit(X, y, Test_Size = 0.3):
    sss = StratifiedShuffleSplit(n_splits=1, test_size=Test_Size, random_state=42)
    _ = sss.get_n_splits(X, y)
    for train_index, test_index in sss.split(X, y):
        # X
        if isinstance(X, pd.DataFrame):
            X_train, X_test = X.loc[train_index], X.loc[test_index]
        else:
            X_train, X_test = X[train_index], X[test_index]
        # y    
        if isinstance(y, pd.Series):
            y_train, y_test = y[train_index], y[test_index]
        else:
            y_train, y_test = y[train_index], y[test_index]
    del sss
    return X_train, y_train, X_test, y_test

X_train, y_train, X_test, y_test = HD_StratifiedShuffleSplit(X_std, y)

from HD_DeepLearning import Train_Test_Dist  
PD.update(dict(column_widths=[0.3, 0.3, 0.3], tablecolumnwidth = [0.2, 0.4], height = 550, legend_title = Target))

Train_Test_Dist(X_train, y_train, X_test, y_test, PD, Labels_dict)

Modeling: Tensorflow DNNClassifier

Here, we use the Tensorflow Linear classifier model.tf.estimator.DNNClassifier.

Input Function The input_function defines how data is converted to a tf.data.Dataset that provides the input pipeline.

• features - A Python dictionary in which:
  • Each key is the name of a feature.
  • Each value is an array containing all of that feature's values.
• label - An array containing the values of the label for every example.

def input_fn(features, labels, training=True, batch_size=256):
    """An input function for training or evaluating"""
    # Convert the inputs to a Dataset.
    dataset = tf.data.Dataset.from_tensor_slices((dict(features), labels))

    # Shuffle and repeat if you are in training mode.
    if training:
        dataset = dataset.shuffle(1000).repeat()
    
    return dataset.batch(batch_size)

Moreover, an estimator model consists of two main parts, feature columns, and a numeric vector. Feature columns provide explanations for the input numeric vector. The following function separates categorical and numerical columns (features)and returns a descriptive list of feature columns.

import tensorflow as tf
def Feat_Columns(Inp):
    Temp = Inp.dtypes.reset_index(drop = False)
    Temp.columns = ['Features', 'Data Type']
    Temp['Data Type'] = Temp['Data Type'].astype(str)
    # Numeric_Columns
    Numeric_Columns = Temp.loc[Temp['Data Type'].isin(['int64', 'int32', 'float64', 'float32']),'Features'].tolist()
    # Categorical_Columns
    Categorical_Columns = Temp.loc[Temp['Data Type'] == 'object','Features'].tolist()
        
    # Feature Columns
    feature_columns = []
    if len(Categorical_Columns)>0:
        for feature_name in Categorical_Columns:
            vocabulary = Inp[feature_name].unique()
            feature_columns.append(tf.feature_column.categorical_column_with_vocabulary_list(feature_name, vocabulary))
    if len(Numeric_Columns)>0:
        for feature_name in Numeric_Columns:
            feature_columns.append(tf.feature_column.numeric_column(feature_name))
    return feature_columns

my_feature_columns = Feat_Columns(X_std)

Estimator using the Default Optimizer

from IPython.display import clear_output
tf.keras.backend.clear_session()
IT = int(5e2)
classifier = tf.estimator.DNNClassifier(feature_columns = my_feature_columns,
                                        # Hidden layers.
                                        hidden_units = [1024, 512, 256],
                                        # The model must choose between 3 classes.
                                        n_classes = len(Labels_dict))
classifier.train(input_fn=lambda: input_fn(X_train, y_train, training=True), max_steps = IT)
result = classifier.evaluate(input_fn=lambda: input_fn(X_test, y_test, training=False))
clear_output()
display(pd.DataFrame(result, index = ['']).round(4))

	accuracy	average_loss	loss	global_step
	0.9815	0.2498	0.2498	500

ROC Curves

from HD_DeepLearning import ROC_Curve

# converting y_test to categorical
y_test_cat = tf.keras.utils.to_categorical(y_test, num_classes = len(Labels_dict), dtype='float32')

pred_dicts = list(classifier.predict(input_fn=lambda: input_fn(X_test, y_test, training=False)))
clear_output()
probs = np.array([pred['probabilities'] for pred in pred_dicts])    
ROC_Curve(y_test_cat, probs, n_classes = len(Labels_dict), FS = 8)

Confusion Matrix

The confusion matrix allows for visualization of the performance of an algorithm. Note that due to the size of data, here we don't provide a Cross-validation evaluation. In general, this type of evaluation is preferred.

from sklearn import metrics
from HD_DeepLearning import Confusion_Mat

# Train
pred_dicts = list(classifier.predict(input_fn=lambda: input_fn(X_train, y_train, training=False)))
clear_output()
probs = np.array([pred['probabilities'] for pred in pred_dicts])
y_pred = np.argmax(probs, axis = 1).reshape(-1,1)
Reports_Train = pd.DataFrame(metrics.classification_report(y_train, y_pred, target_names=list(Labels_dict.values()),
                                                           output_dict=True)).T
CM_Train = metrics.confusion_matrix(y_train, y_pred)
# Test
pred_dicts = list(classifier.predict(input_fn=lambda: input_fn(X_test, y_test, training=False)))
clear_output()
probs = np.array([pred['probabilities'] for pred in pred_dicts])
y_pred = np.argmax(probs, axis = 1).reshape(-1,1)
Reports_Test = pd.DataFrame(metrics.classification_report(y_test, y_pred, target_names=list(Labels_dict.values()),
                                                          output_dict=True)).T
CM_Test = metrics.confusion_matrix(y_test, y_pred)

Reports_Train = Reports_Train.reset_index().rename(columns ={'index': 'Train Set'})
Reports_Test = Reports_Test.reset_index().rename(columns ={'index': 'Test Set'})
                                                 
display(Reports_Train.style.hide(axis='index').set_properties(**{'background-color': 'HoneyDew', 'color': 'Black'}).\
        set_properties(subset=['Train Set'], **{'background-color': 'SeaGreen', 'color': 'White'}))
display(Reports_Test.style.hide(axis='index').set_properties(**{'background-color': 'Azure', 'color': 'Black'}).\
        set_properties(subset=['Test Set'], **{'background-color': 'RoyalBlue', 'color': 'White'}))

PD = dict(FS = (10, 6), annot_kws = 14, shrink = .6, Labels = list(Labels_dict.values()))
Confusion_Mat(CM_Train, CM_Test, PD = PD, n_splits = None)

Train Set	precision	recall	f1-score	support
Class_0	1.000000	1.000000	1.000000	41.000000
Class_1	1.000000	0.980000	0.989899	50.000000
Class_2	0.970588	1.000000	0.985075	33.000000
accuracy	0.991935	0.991935	0.991935	0.991935
macro avg	0.990196	0.993333	0.991658	124.000000
weighted avg	0.992173	0.991935	0.991955	124.000000

Test Set	precision	recall	f1-score	support
Class_0	1.000000	1.000000	1.000000	18.000000
Class_1	1.000000	0.952381	0.975610	21.000000
Class_2	0.937500	1.000000	0.967742	15.000000
accuracy	0.981481	0.981481	0.981481	0.981481
macro avg	0.979167	0.984127	0.981117	54.000000
weighted avg	0.982639	0.981481	0.981554	54.000000

Estimator using an Optimizer with a Learning Rate Decay

In this classification, the learning rate of your optimizer changes over time.

tf.keras.backend.clear_session()
classifier = tf.estimator.DNNClassifier(feature_columns=my_feature_columns,
            # Hidden layers.
            hidden_units= [1024, 512, 256],
            # The model must choose between 3 classes.
            n_classes=len(Labels_dict),
            optimizer=lambda: tf.keras.optimizers.Adam(learning_rate=tf.compat.v1.train.exponential_decay(learning_rate=0.1,
            global_step=tf.compat.v1.train.get_global_step(), decay_steps=IT,decay_rate=0.96)))
#
classifier.train(input_fn=lambda: input_fn(X_train, y_train, training=True), max_steps= IT)
result = classifier.evaluate(input_fn=lambda: input_fn(X_test, y_test, training=False))
clear_output()
display(pd.DataFrame(result, index = ['']).round(4))

	accuracy	average_loss	loss	global_step
	0.9815	224.856201	224.856201	500

ROC Curves

# converting y_test to categorical
y_test_cat = tf.keras.utils.to_categorical(y_test, num_classes = len(Labels_dict), dtype='float32')

pred_dicts = list(classifier.predict(input_fn=lambda: input_fn(X_test, y_test, training=False)))
clear_output()
probs = np.array([pred['probabilities'] for pred in pred_dicts])    
ROC_Curve(y_test_cat, probs, n_classes = len(Labels_dict), FS = 8)

Confusion Matrix

# Train
pred_dicts = list(classifier.predict(input_fn=lambda: input_fn(X_train, y_train, training=False)))
clear_output()
probs = np.array([pred['probabilities'] for pred in pred_dicts])
y_pred = np.argmax(probs, axis = 1).reshape(-1,1)
Reports_Train = pd.DataFrame(metrics.classification_report(y_train, y_pred, target_names=list(Labels_dict.values()),
                                                           output_dict=True)).T
CM_Train = metrics.confusion_matrix(y_train, y_pred)
# Test
pred_dicts = list(classifier.predict(input_fn=lambda: input_fn(X_test, y_test, training=False)))
clear_output()
probs = np.array([pred['probabilities'] for pred in pred_dicts])
y_pred = np.argmax(probs, axis = 1).reshape(-1,1)
Reports_Test = pd.DataFrame(metrics.classification_report(y_test, y_pred, target_names=list(Labels_dict.values()),
                                                          output_dict=True)).T
CM_Test = metrics.confusion_matrix(y_test, y_pred)

Reports_Train = Reports_Train.reset_index().rename(columns ={'index': 'Train Set'})
Reports_Test = Reports_Test.reset_index().rename(columns ={'index': 'Test Set'})
                                                 
display(Reports_Train.style.hide(axis='index').set_properties(**{'background-color': 'HoneyDew', 'color': 'Black'}).\
        set_properties(subset=['Train Set'], **{'background-color': 'SeaGreen', 'color': 'White'}))
display(Reports_Test.style.hide(axis='index').set_properties(**{'background-color': 'Azure', 'color': 'Black'}).\
        set_properties(subset=['Test Set'], **{'background-color': 'RoyalBlue', 'color': 'White'}))

PD = dict(FS = (10, 6), annot_kws = 14, shrink = .6, Labels = list(Labels_dict.values()))
Confusion_Mat(CM_Train, CM_Test, PD = PD, n_splits = None)

Train Set	precision	recall	f1-score	support
Class_0	1.000000	1.000000	1.000000	41.000000
Class_1	1.000000	1.000000	1.000000	50.000000
Class_2	1.000000	1.000000	1.000000	33.000000
accuracy	1.000000	1.000000	1.000000	1.000000
macro avg	1.000000	1.000000	1.000000	124.000000
weighted avg	1.000000	1.000000	1.000000	124.000000

Test Set	precision	recall	f1-score	support
Class_0	1.000000	1.000000	1.000000	18.000000
Class_1	0.954545	1.000000	0.976744	21.000000
Class_2	1.000000	0.933333	0.965517	15.000000
accuracy	0.981481	0.981481	0.981481	0.981481
macro avg	0.984848	0.977778	0.980754	54.000000
weighted avg	0.982323	0.981481	0.981378	54.000000

---

References

1. Regression analysis Wikipedia page
2. Tensorflow tutorials
3. Online machine learning Wikipedia page
4. Learning rate Wikipedia page
5. S. Aeberhard, D. Coomans and O. de Vel, Comparison of Classifiers in High Dimensional Settings, Tech. Rep. no. 92-02, (1992), Dept. of Computer Science and Dept. of Mathematics and Statistics, James Cook University of North Queensland. (Also submitted to Technometrics).
6. S. Aeberhard, D. Coomans and O. de Vel, “THE CLASSIFICATION PERFORMANCE OF RDA” Tech. Rep. no. 92-01, (1992), Dept. of Computer Science and Dept. of Mathematics and Statistics, James Cook University of North Queensland. (Also submitted to Journal of Chemometrics).

---